Spatio-Temporal Video Analysis and the 3D Shearlet Transform

Abstract The automatic analysis of the content of a video sequence has captured the attention of the computer vision community for a very long time. Indeed, video understanding, which needs to incorporate both semantic and dynamic cues, may be trivial for humans, but it turned out to be a very complex task for a machine. Over the years the signal processing, computer vision, and machine learning communities contributed with algorithms that are today effective building blocks of more and more complex systems. In the meanwhile, theoretical analysis has gained a better understanding of this multifaceted type of data. Indeed, video sequences are not only high dimensional data, but they are also very peculiar, as they include spatial as well as temporal information which should be treated differently, but are both important to the overall process. The work of this thesis builds a new bridge between signal processing theory, and computer vision applications. It considers a novel approach to multi resolution signal processing, the so-called Shearlet Transform, as a reference framework for representing meaningful space-time local information in a video signal. The Shearlet Transform has been shown effective in analyzing multi-dimensional signals, ranging from images to x-ray tomographic data. As a tool for signal denoising, has also been applied to video data. However, to the best of our knowledge, the Shearlet Transform has never been employed to design video analysis algorithms. In this thesis, our broad objective is to explore the capabilities of the Shearlet Transform to extract information from 2D+T-dimensional data. We exploit the properties of the Shearlet decomposition to redesign a variety of classical video processing techniques (including space-time interest point detection and normal flow estimation) and to develop novel methods to better understand the local behavior of video sequences. We provide experimental evidence on the potential of our approach on synthetic as well as real data drawn from publicly available benchmark datasets. The results we obtain show the potential of our approach and encourages further investigations in the near future

Space-Time Signal Analysis and the 3D Shearlet Transform

In this work, we address the problem of analyzing video sequences by representing meaningful local space\ue2\u80\u93time neighborhoods. We propose a mathematical model to describe relevant points as local singularities of a 3D signal, and we show that these local patterns can be nicely highlighted by the 3D shearlet transform, which is at the root of our work. Based on this mathematical framework, we derive an algorithm to represent space\ue2\u80\u93time points which is very effective in analyzing video sequences. In particular, we show how points of the same nature have a very similar representation, allowing us to compute different space\ue2\u80\u93time primitives for a video sequence in an unsupervised way

Detecting spatio-temporally interest points using the shearlet transform

In this paper we address the problem of detecting spatio-temporal interest points in video sequences and we introduce a novel detection algorithm based on the three-dimensional shearlet transform. By evaluating our method on different application scenarios, we show we are able to extract meaningful spatio-temporal features from video sequences of human movements, including full body movements selected from benchmark datasets of human actions and human-machine interaction sequences where the goal is to segment drawing activities in smaller action primitives

Good Practices on Hand Gestures Recognition for the Design of Customized NUI

n/

Analysis of the qualities of human movement in individual action

The project was organized as a preliminary study for Use Case #1 of the Horizon 2020 Research Project \u201cDance in the Dark\u201d (H2020 ICT Project n.645553 - http://dance.dibris.unige.it). The main objective of the DANCE project is to study and develop novel techniques and algorithms for the automated measuring of non-verbal bodily expression and the emotional qualities conveyed by human movement, in order to enable the perception of nonverbal artistic whole-body experiences to visual impaired people. In the framework of the eNTERFACE \u201915 Workshop we investigated methods for analyzing human movements in terms of expressive qualities. When analyzing an individual action we were mainly concentrating on the quality of motion and on elements suggesting different emotions. We developed a system to automatically extract several movement features and transfer them to the auditory domain through interactive sonification. We performed an experiment with 26 participants and collected a dataset made of video and audio recordings plus accelerometer data. Finally, we performed a perception study through questionnaires, in order to evaluate and validate the system. As real time application of our system we developed a game named \u201dMove in the Dark\u201d, which has been presented in the Mundaneum Museum of Mons, Belgium and Festival della Scienza, Genova, Italy (27 November 2015)

Local spatio-temporal representation using the 3D shearlet transform

In this work we address the problem of analyzing video sequences and representing meaningful space-time points of interest. We base our work on the 3D shearlet transform. In particular, we exploit the relation between coefficients with similar shearings to build a local representation which turns out to be really informative to understand the local spatio-temporal characteristics of the points that we are considering

Rock Art Interpretation within Indiana MAS

This paper presents the first results achieved within the Indiana MAS project funded by Italian Ministry for Education, University and Research, MIUR. We discuss how the AgentSketch holon belonging to the Indiana MAS has been extended to cope with images, besides hand drawn sketches, and has been tested in the domain of Mount Bego’s prehistoric rock art (southern French Alps). The way Indiana MAS holons cooperate in order to provide correct interpretations of ambiguous shapes is discussed by means of an example based on hypotheses recently advanced by archaeologists

A holonic multi-agent system for sketch, image and text interpretation in the rock art domain

10sireservedThis paper presents the architecture of a holonic multi-agent system for rock art interpretation and discusses the results achieved within the "Indiana MAS" project. We show how the AgentSketch and the ImageRec holons belonging to the Indiana MAS, able to cope with hand drawn sketches and images respectively, have been tested in the domain of Mount Bego's prehistoric rock art (southern French Alps), and how the Ma-nent agent-based framework for the seamless integration of Digital Libraries has been plugged into Indiana MAS to provide text classification, as well as multilingual access to structured repositories. The way Indiana MAS holons cooperate in order to provide correct interpretations of ambiguous shapes is discussed by means of an example based on hypotheses recently advanced by archaeologists. © 2014.mixedMascardi, V; Briola, D; Locoro, A; Grignani, D; Deufemia, V; Paolino, L; Bianchi, N; de Lumley, H; Malafronte, D; Ricciarelli, AMascardi, V; Briola, D; Locoro, A; Grignani, D; Deufemia, V; Paolino, L; Bianchi, N; de Lumley, H; Malafronte, D; Ricciarelli,